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The Keys to Implementing New Technology 
While Providing Evidence that Technology is Successful 
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The Scientific Method has broader application to any situation 
where the steps of gathering observable, measurable evidence, 
using good objective reasoning to evaluate outcomes, and then 
repeating the cycle are warranted. 


P roving that technology works... 
It is not as simple as proving 
that a new vendor for art sup- 
plies is more cost effective. Technol- 
ogy effectiveness requires both the 
right software and the right imple- 
mentation. Just having the software is 
not enough. Proper planning, training, 
leadership, support, pedagogy, and 
software use — along with many other 
factors — will determine whether the 
technology works. When so many fac- 
tors can affect the outcome, we need 
a proven protocol, as almost every 
administrator, principal, and teacher 
will agree. But still many technology 
implementations are less than suc- 
cessful. As a scientist and a teacher, I 
believe that we can borrow from the 
Scientific Method and from some 
common-sense approaches to in- 
crease our chances of success. 

Most people think of the Scientific 
Method as a set of five steps to use 
when creating a science experiment. 
But it has broader application to any 
situation where the steps of gathering 
observable, measurable evidence, us- 
ing good objective reasoning to evalu- 
ate outcomes, and then repeating the 
cycle are warranted. Such a process is 
essential to show the strengths, weak- 
nesses, and return on investment for 
any program a school chooses to use. 

The Software and Information 
Industry Association has released a 
checklist that includes 10 essential 
concepts that underlie all effective 
implementations. ( Editor's Note: A list 
of the 10 concepts begins on page 14. 
The list is available at http://www.siia. 
net/ education/ pubs/ pp_Checklist.pdf. ) 
These concepts embody the things I 
try to instill into everyone I help with 
implementation. In addition, SIIA 
also has a toolkit to help schools work 
through the process of an implementa- 
tion of any software, administrative or 
instructional (http://www.siia.net/edu- 
cation / foreducators/ toolkit_0407.pdf) . 

While we can list the concepts, it’s 
easiest to see how they apply using a 


story. This story involves a fictitious 
school district, Valley District, which 
has four elementary schools, two 
middle schools, and one high school. 
As the story begins, two fourth grade 
teachers at one of the elementary 
schools discover a new reading soft- 
ware program. The teachers feel 
strongly that this program would be 
better than the one currently being 
used that dates from the 1980s. They 
believe this new program would re- 
ally reach the students, getting them 
further engaged in their own learning. 
So, the teachers (Ms. Crum and Mr. 
Pappelwith) approach their principal 
about switching to it. Principal Leaven 
likes the idea but reminds them that 
the school board is demanding evi- 
dence that something is truly a benefit 
to the students of Valley District be- 
fore expenditures will be authorized. 
Ms. Crum and Mr. Pappelwith imme- 
diately begin talking about what they 
can do to meet this requirement. 

Stage 1 : Create the Hypothesis 

Principal Leaven, Ms. Crum, and Mr. 
Pappelwith meet with Mrs. White, the 
district elementary school director of 
curriculum and instruction. The four 
decide to move forward. They know 
they need to develop a plan to deter- 
mine if the new reading program will 
help the students learn more effec- 
tively than the current program. 

Stage 2: Planning the Experiment 

They recognize that planning is not as 
simple as one person sitting at a desk 
and mapping it all out; it will take a 
committee of the right people asking 
the right questions. They decide to 
solicit questions from others to make 
sure they haven’t missed any impor- 
tant considerations. Once the ques- 


tions are on the table, the committee 
will need to come to an agreement 
about how to handle those questions 
and keep good records of the process. 

Another important part of the plan- 
ning stage is leadership (SIIA concept 
5). The principal and teachers are 
aware of the need to make sure that 
the project’s leaders have the ability 
and willingness to change things as 
they need to be changed. In a district 
setting this could be several people — 
including a district-level leader and a 
school leader. 

So our Valley District Reading 
Program committee begins asking 
questions. They each talk about the 
proposed program in various meet- 
ings. Who should we involve in this 
process? What does effective mean? 
How do we define it? What teachers/ 
classes should be involved? How is it 
recommended to be used? How can 
we integrate it into the class? What 
can we use to evaluate the results? 

These lead to more questions. How 
big do we need to start? Who are the 
stakeholders? What group can we use 
as a study group? What group can we 
use for a control group? Is this the 
right group to be involved in the plan- 
ning? Who will be strong leaders for 
this project? 

Because Ms. Crum and Mr. Pap- 
pelwith are both teachers in the same 
school and the same grade, they 
are the obvious school leaders and 
those who will be most committed 
to working through any difficulties 
that they run into. The district de- 
cides to implement the new software 
in one of the two classes, which will 
allow them to compare the two ap- 
proaches. Although the experiment is 
not a strict random control and study 
group, it will allow for a comparison 
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The scientific method is often referred 
to as having five steps. They are: 

(or state something to 
investigate). 

(or invent an 

explanation) based on already- 
known facts. 

about how to 

appropriately test the hypothesis. 

using planned experiments. 

until there are no discrepancies 
between experiment and observations. 


with known parameters. As long as 
they don t make claims beyond those 
parameters, the setup should allow for 
a good initial investigation and may 
provide information to move forward. 
It also provides an effective way to 
make adjustments. It is easier to ad- 
just with a smaller number of students, 
figure out the best approach with one 
class, and then expand to others. 

The district curriculum and instruc- 
tion director is also a good choice 
for leadership at the district level. 
Because this is a software program, 
we need to get the IT department 
involved and make sure we com- 
municate with the software vendor. 

We don t want any technological 
problems to interfere with the project. 
The people that need to stay closely 
informed are the other teachers in the 
district, the principal, the students, 
and parents. 

The project team agrees that the 
communications to the stakeholders 


must cover why they will be using it 
with one class and not the other. The 
students, parents, and other teachers 
need to see that the plan is well con- 
ceived to demonstrate the programs 
effectiveness by minimizing the num- 
ber of things that are being changed. 
As a school and a district, the team 
will be able to use this project to 
broaden the education of the students 
about good decision making in life. 

Now that we have a broad-stroke 
plan and have determined who the 
stakeholders are (SIIA concept 3), we 
can address some other questions, 
such as what effective means and how 
we will evaluate the program (SIIA 
concept 4). Regardless of what the 
evaluation measures are, the team has 
to be prepared to look objectively at 
what the evaluation means. 

As it turns out in Valley School 
District, records are plentiful. The 
students at every grade are assessed by 
a district exam at the beginning and 
end of each year as well as by state 
exams. Test scores and other student 
data can be compared between the 
two classes. We can also add more 
subjective data such as student and 
teacher surveys. For each of the meth- 
ods determined to be important for 
evaluation methods, the planning 
committee wisely determines criteria 
that they need to see reached in order 
for the new software implementation 
to be considered successful. Knowing 
ahead of time what the criteria are 
will prevent the results from clouding 


judgment when evaluating the suc- 
cesses of the program, and will help 
determine which results are most 
important to consider, particularly 
because this study will only provide 
a comparison at one school without a 
strict random assigned control. 

When it comes time to install the 
software, the team looks carefully at 
the IT infrastructure of the school 
(SIIA concept 6: environment and 
equipment must match the require- 
ments). Close communication with 
the IT department and the software 
vendor will be critical to the experi- 
ence. This ensures that the computers 
are appropriate for the design specs, 
that access is appropriate for the soft- 
ware, and that the teachers follow the 
software designers guidelines. Mak- 
ing sure that the software is “used as 
recommended” adds a level of validity 
to the project and the evaluation. 

Stage 3: Preparation 

Now the project team is ready to be- 
gin training the stakeholders. (SIIA 
concept 7: adequate training) The 
team schedules training for Ms. Crum, 
Mr. Pappelwith, and Ms. White. The 
training enables them to learn as 
much as possible about the software, 
about the project, and what the ex- 
pected outcomes are. Ms. Crum and 
Mr. Pappelwith hold a meeting with 
the other teachers in their school 
and also one with the parents, mak- 
ing everyone aware of what they are 
trying to accomplish. At the training 


The Ten Essentials The following basic concepts underlie all effective software implementation. 

: You must have objectives ; You must plan before 
; before you plan. ! you implement. 

You must involve 
all stakeholders in the 
planning process to some 
degree, and obtain their 
buy-in for performing their 
assigned responsibilities. 

Your plan 

must include an evaluation 
component and criteria for 
determining success. 

You must assign 
an effective leader, 
with appropriate decision- 
making authority, to manage 
the implementation. 


Reprinted from SIIA's Software Implementation Checklist for Educators. 
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they also address student prerequisites 
for the software to be successful (SIIA 
concept 8: prerequisite knowledge 
and skills). This could include com- 
puter use, but also includes what the 
software expects the students to know. 
This will help with the success and the 
full integration into the curriculum 
itself. They know that for the software 
to be effective it needs to integrate 
into what they are doing as seamlessly 
as possible. 

At this point, Valley School District 
is ready to begin the implementation! 

Stage 4: Testing the Predictions 

Documentation throughout the proj- 
ect is important and the implemen- 
tation is no exception. All teachers 
know that it is rare that something 
ever works according to the initial 
plan; adjustments during the pilot and 
recording those adjustments are criti- 
cal to being able to ramp it up later. 
Ms. Crums class is the study class and 
Mr. Pappelwiths is the control, but 
they are not acting as islands. They 
will communicate frequently through- 
out the implementation as things 
get adjusted and new discoveries are 
made; this will help in the ultimate 
success. Ms. Crum works closely with 
Mrs. White to make sure that every- 
thing is going smoothly and any snags 
are handled expeditiously. 

Stage 5: Adjustments 

Now the school year is over, and 
Ms. Crum is very excited to see the 


outcome. She really thought that the 
number of students engaged in the 
material this year was higher, but will 
the evaluation bear witness to that ef- 
fect? How are the scores? What is the 
comparison between the classes at all 
levels? Are there additional adjust- 
ments to be considered? What is next? 

The project team gathers the data, 
and the assessment begins. The study 
results show a modest increase in 
reading scores with the new program. 
Ms. Crum and Mr. Pappelwith did a 
good job of documenting what they 
did. Data on behavior and attendance, 
as well as more subjective survey re- 
sults from the students in both groups, 
show that switching really is worth 
considering. The material does seem 
to engage the students better, and it is 
more relevant to their lives. In addi- 
tion, teacher documentation showed 
it was easier to integrate into the read- 
ing curriculum. All their evaluative 
measures seem to meet or beat their 
previously set criteria. 

After looking at the results, the team 
decides that a larger implementation 
is warranted and they make the rec- 
ommendation to the district, which 
then decides to use this new program 
with two of the four elementary 
schools next year, with the plan to 
expand to the remaining schools the 
next year. 

Each of these implementations will 
need to be monitored like the pilot 
to make sure that as the number of 
variables (such as students and teach- 


ers) increase, that the methodology 
still works. The best implementation 
includes a continual review of effec- 
tiveness. In fact, the team needs to 
bear in mind that the evaluation never 
really ends. That includes making sure 
that the software continues to be used 
as it is recommended and that it is 
integrated into what the teachers are 
doing in the classroom. 

The results can now be shared with 
the stakeholders, used to build the 
next stage of implementation and 
evolution of best practices. The dis- 
trict can proceed with confidence in 
rolling out the new program based on 
evidence within the boundaries of the 
investigation they performed, a clearly 
documented process, and engage- 
ment of all the stakeholders. In short, 
the school has modeled the Scientific 
Method for its students in the choice 
and implementation of the learning 
programs they experience and has 
demonstrated that the software works 
for them in ways that are meaningful 
to everyone. 


Sara M. Sawtelle, PhD, taught 
chemistry and served as an 
instructional technology ad- 
ministrator. As director of 
scientific affairs for Learning 
Enhancement Corporation, 
she conducts research and 
helps clinicians and schools implement Brain- 
Ware Safari, a program that develops the cogni- 
tive skills most critical for learning in an engag- 
ing video-game format. She was a contributing 
author for the SIIAs Software Implementation 
Toolkit for Educators. 




The physical environment 

The staff involved 

The end users of 

The entire implementation 

Each stage of 

and equipment for the 

in the implementation 

the software must have 

must be monitored and the 

implementation must 

implementation must match 
the requirements for which 
the software was designed. 

must receive adequate 
training in all aspects 
of the software’s use. 

the prerequisite 
knowledge and skills 
specified for the 
product’s use. 

resulting issues resolved. 

be informed by the 
evaluation results from 
the previous stage. 
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